Bradley/stackbench test - #5766
Draft
bradleyshep wants to merge 467 commits into
Draft
Conversation
A provider 429 or 529 is a transient account condition, not a session defect. The recovery loop retried it immediately, so every resume burned in milliseconds while the account stayed throttled, and the campaign's declared retry then repeated the same futile sequence. Seven executions of the first qualified 3x3 cohort died this way. Throttle interruptions now wait on an escalating schedule under their own time budget (STACK_BENCH_PROVIDER_THROTTLE_MAX_WAIT_MINUTES, default 300) instead of consuming the bounded interruption retries, resume the exact paid session when one exists, and restart from the existing files when the first request itself was rejected. Waits are logged, recorded in the session interruptions, and reported in run setup so duration comparisons cannot blame a backend for the account's usage window. The adapter deadline also rises above one full coding session plus the throttle budget: the old 75-minute value sat below the 123-minute session bound, so the supervisor's SIGTERM pended through the synchronous container call and reported live sessions as failures. Adapter contract 1.11.0 -> 1.12.0.
Campaign run validation canonicalized selection fields straight from the artifact, so a run that omitted one crashed with 'definition plan at $ is not canonical JSON data' instead of naming the diverging field. A missing field now registers as an ordinary identity mismatch. Also repairs the test debt left by the modular example migration: the level fixtures now carry the exact planned selection a valid run.json records, and the condition-version expectation matches the example.
Four operations the grader calls by fixed name were never stated to the builder: updateCartQuantity (PATCH /api/cart/:itemId) at L1, and adminTransferStock, adminChangePrice, and cancelOrder with their routes at L2. Apps passed those checks only when they guessed the reference's name — a lottery REST convention wins more often than reducer naming, so it biased L2 against SpacetimeDB. In the running 3x3 cohort, spacetime attempts enforce authorization correctly (530 refusals where the reducer exists) but lose 201a/201b to a 404 on a name they were never told. New draft releases state the names the way L1 already states its six: the cart feature text names updateCartQuantity, and the three L2 packs gain pack-owned testing-call tables. Recipe diff proves scoring, fixtures, and execution are byte-identical; only task fragments were added, and the L2 calibration is correctly invalidated pending a fresh qualification of l1-modular@2.5.0 and l2-standard@1.6.0. A 404 on a named action now also names the required operation in the failure text, so a repair round can create the missing endpoint instead of chasing authorization. Verdict logic is unchanged.
The reseed/restart probe reads GET /api/items and requires a JSON object with an items array, but no prompt stated the endpoint or its shape - a server-based app returning a bare array or routing its catalogue elsewhere fails as an application startup-seeding failure it was never told how to avoid. Same undeclared-surface class as the named testing operations; found by completing that audit. The draft l1-modular@2.5.0 now states the endpoint as recipe framing; SpacetimeDB apps are exempt because their probe goes through the module.
A postgres attempt seeded only on first install, came back empty after the harness's database reset, and scored 0/58 without grading. The task now states the behavior the reseed probe verifies: the server seeds its starting data at startup whenever it is missing. Draft 2.5.0 fragment only; qualified releases untouched.
The landing table led with cost to correct - the secondary metric - and rendered a completed campaign's status as an empty cell while stale history shouted for attention. Each stack cell now leads with the first-build score, shows the repaired score and median spend beside it, and stars a cost the campaign's own detail view would refuse to compare. Completed says so. History that needed attention days ago archives out of the default view behind a toggle, campaign keys distinguish identically titled rows, and running campaigns pin to the top with a per-attempt strip: phase, latest score, elapsed time, spend so far. Refresh now reconciles keyed rows in place instead of replacing the table, so an element under the pointer survives the poll; the whole row is the click target. Campaign detail is URL-addressable (#campaign=<key>), opens facts an operator previously read plan.json for (agent, model, exact recipes, pinned image digests), and each collapsed attempt answers first -> final, rounds, duration, and spend, with the still-failing checks as chips. The model distinguishes a first grade that aborted before scoring (seed probe, startup failure) from a scored zero - an app the grader never scored did not score 0 percent - and level failure lists now use the authoritative final appFailures instead of falling back to first-build misses on repaired attempts. Levels gain durations; plans gain identity facts.
Medians answer which stack won; an operator watching a cohort needs the nine runs. A campaign row now expands in place into one row per attempt - stack and repetition, state or live phase, level reached, first and latest score, rounds, elapsed time, spend - updating on every poll. Running campaigns start expanded, the choice survives refreshes, and the title button still opens the evidence modal. Replaces the running-only chip strip, which compressed the same facts into prose.
The page led with a history table; the operator's questions - is it healthy, how far along, what just happened, is anything stuck - had no surface. Three zones now answer them in order. Now: each running campaign is a card with attempts done, burn against the declared ceiling, and elapsed time; each attempt is a lane with a score bar, live phase, a score-per-round sparkline, and an event line. Two failure smells the operator previously diagnosed from logs are automatic: a trailing run of identical grades flags 'flat N rounds', and a running attempt whose output file has gone quiet flags 'no output Nm'. Scores flash on change and a liveness tick shows the last refresh while anything runs. Latest verdict: the most recent completed campaign renders as a dot-and-whisker row per stack on a shared first-build axis, repaired score and median spend beside it, with the aborted-grade and exclusion footnotes attached to the claim they qualify. History: the table, demoted below both, without running campaigns. Model: parseRunProgress keeps the whole grade series instead of only first and latest, and attempts expose when their log last changed.
The history table forced three stack columns onto every campaign, so a single-stack smoke run rendered as a comparison with two holes and a starred median. History's job is identification: each row now carries the participating stacks beside the key, one adaptive headline showing first->repaired only for stacks that actually graded, and the campaign's total burn - always well-defined - instead of per-stack medians whose comparability belongs to the verdict and detail views. The detail view stops being a floating window: it is a full-height right side panel that closes on outside click or Escape, keeping its deep link. A top load bar reflects requests actually in flight.
The adaptive headline shipped without its layout, so per-stack figures fused into unreadable run-on text. Each history row's result is now a two-column block - stack labels, aligned first->repaired figures - with the campaign's total burn on its own line.
Three redesigns tried to make a summary stat legible inside a list cell; the right answer was that it does not belong there. A history row now identifies the campaign - title, key, participating stacks, status, recency - and nothing else. The numbers live where they can be read honestly: the verdict card, the row's per-attempt expansion, and the detail panel.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of Changes
This PR adds Stack Bench, a reproducible benchmark for coding agents building applications against SpacetimeDB, PostgreSQL, and MongoDB.
Benchmark definitions and grading
passed,failed,inconclusive, orharness_failure.Ecommerce benchmark
Runs, repairs, and campaigns
CLI, dashboard, and appliance
SpacetimeDB TypeScript SDK
TableReffrom an anonymous readonly type alias to a readonly interface with the same members.TableRefis used incorrectly.ConnectionManagerreconnects to reuse the identity token previously issued by the server.TableRefdiagnostics.SpacetimeDB CLI
spacetime devto retain configuration changes made during direct-login publishing.SpacetimeDB skills
API and ABI breaking changes
This PR changes the public Rust signature of:
spacetimedb_cli::publish::exec_from_entryThe function now accepts
&mut Configinstead of takingConfigby value. External Rust callers must pass a mutable reference. The API-breaking label should be applied unless the previous entry point is preserved through a compatibility wrapper.The exported TypeScript
TableRefdeclaration changes from a type alias to an interface while retaining the same readonly member contract.Automatic TypeScript client reconnection also changes behavior by presenting the identity token from the previous successful connection.
Expected complexity level and risk
3 / 5
Most of the implementation is isolated under
tools/stack-bench. The principal risks are benchmark correctness, evidence integrity, concurrent resource isolation, interrupted-run recovery, Docker cleanup, and credential handling.The TypeScript reconnect change and
spacetimedb-clihelper signature are the only changes outside the benchmark and skill directories.Testing
TableRefregression tests: 4/4 passed.cargo check -p spacetimedb-cli.exec_from_entryRust signature.